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DETAILED ACTION 
Response to Arguments 

Applicant's arguments with respect to claims 1 - 45 have been considered but are moot 
since these claims have been cancelled. 

Applicant argues that Harrington does not discloses "producing the augmented reality 
video by rendering the three-dimensional image data model of the product superimposed on the 
moving model plane in that video data and having the pose of the moving model plane". 
Harrington further discloses the unprojection module 82 that projects three-dimension object 
onto a two-dimensional viewing plane, where a two dimensional image results from the three 
dimensional object and the unprojection step comprises the mathematical calculation comprising 
taking a two-dimensional image and determining what three-dimensional object could have 
created that two-dimensional image with respect to what the object can be and where it could 
have been positioned and the subject invention utilizes known dimensions with regard to the 
camera system and viewing plane, in combination with assumed characteristics of the piece of 
paper (column 7, lines 28 - 53), 

The specification objections have been withdravm due to applicant's amendments. 

The abstract objection has been withdrawn due to applicant's amendments. 

The 35 U.S.C. 1 12 rejections have been withdrawn due to the cancellation of claims 1 - 

45. 
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Claim Rejections - 35 USC § 102 
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the 
basis for the rejections under this section made in this Office action: 
A person shall be entitled to a patent unless - 

(e) the invention was described in (1) an application for patent, published under section 122(b), by another filed 
in the United States before the invention by the applicant for patent or (2) a patent granted on an application for 
patent by another filed in the United States before the invention by the applicant for patent, except that an 
international application filed under the treaty defined in section 351(a) shall have the effects for purposes of this 
subsection of an application filed in the United States only if the international application designated the United 
States and was publ ished under Article 21(2) of such treaty in the English language. 

(f) he did not himself invent the subject matter sought to be patented. 



Claims 46 - 48, 52 - 56, and 60 are rejected under 35 U.S.C. 102(e) as being anticipated 
by Harrington (TJS 6,898,307). 

Regarding to independent claim 46, Harrington discloses a camera 12 that captures the 
normal field of view of the user of the headset 10 that is substantially what the user would see 
without the headset 10 (column 4, lines 4 - 24) and the analog signal from the camera 12 is 
converted to a digitized frame that can be stored in the computer 1 8 for analysis (column 4, lines 
25 - 38) and the computer 18 must interpret the captured video frames to locate a piece of paper 
within the camera view (column 4, lines 38 - 67 and column 5, lines 1 - 34) and the preselected 
movable real object comprises a reference panel such as a screen, tablet or piece of paper and the 
identifying includes recognizing a comer of the panel (claim 2), which reads on the claimed 
providing video data comprising images of a moving model plane having markers. Harrington 
further discloses the unprojection module 82 that projects three-dimension object onto a two- 
dimensional viewing plane, where a two dimensional image results from the three dimensional 
object (column 7, lines 28 - 53), which reads on the claimed providing a three-dimensional 
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image data model of a product. Harrington also discloses the unprojection step comprises the 
mathematical calculation comprising taking a two-dimensional image and determining what 
three-dimensional object could have created that two-dimensional image with respect to what the 
object can be and where it could have been positioned and the subject invention utilizes known 
dimensions with regard to the camera system and viewing plane, in combination with assumed 
characteristics of the piece of paper (column 7, lines 28 - 53), which reads on the claimed 
determining a pose of the moving model plane according to the markers in the video data and the 
claimed determining an image correspondence between the moving model plane having markers 
and the three-dimensional image data model. Harrington discloses the camera provide the 
computer with the same view that the user sees so that the synthetic image is displayed or 
superimpose in a manner for comfortable and convenient interaction between the real reference 
item within the field of view that the user may be holding and manipulating (column 4, lines 4 - 
37), which reads on the claimed producing the augmented reality video by rendering the three- 
dimensional image data model of the product superimposed on the moving model plane in the 
video data. 

Regarding independent claim 54, claim 54 is similar in scope as to claim 46, thus the 
rejection for claim 46 hereinabove is applicable to claim 54. 

Regarding dependent claim 47 and 55, Harrington discloses the user could select as the 
application image a particular internet browser engine which, based upon the located orientation 
of the piece of paper, would be transformed to exactly match the detected orientation of the piece 
of paper so that the displayed image provides the illusion of the web page actually appearing on 
the blank piece of paper being held by the user and once the position of the paper page is 
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determined, information can be used to correctly distort the application's image so it overlays the 
paper in the eyes of the user (column 5, lines 4-35) and the unprojection step comprises the 
mathematical calculation comprising taking a two-dimensional image and determining what 
three-dimensional object could have created that two-dimensional image with respect to what the 
object can be and where it could have been positioned and the subject invention utilizes known 
dimensions with regard to the camera system and viewing plane, in combination with assumed 
characteristics of the piece of paper (column 7, lines 28 - 53), which reads on the claimed 
rendering of the three-dimensional image data model has substantially the pose of the moving 
model plane. 

Regarding dependent claims 48 and 56, Harrington discloses the translation, rotation, 
scaling and perspective operations can be applied to the image using standard image processing 
methods, or specialized textual mapping hardware may be employed for improving processing 
speed (column 5, lines 26 - 35) and the unprojection step comprises the mathematical calculation 
comprising taking a two-dimensional image and determining what three-dimensional object 
could have created that two-dimensional image with respect to what the object can be and where 
it could have been positioned and the subject invention utilizes known dimensions with regard to 
the camera system and viewing plane, in combination with assumed characteristics of the piece 
of paper (column 7, lines 28 - 53), which reads on the claimed scaling the three-dimensional 
image data model to the model plane according to the markers. 

Regarding dependent claims 52 and 60, Harrington discloses the user could select as the 
application image a particular internet browser engine which, based upon the located orientation 
of the piece of paper, would be transformed to exactly match the detected orientation of the piece 
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of paper so that the displayed image provides the illusion of the web page actually appearing on 
the blank piece of paper being held by the user (column 5, lines 15 - 22), which reads on the 
claimed video data includes images of a human manipulating the pose of the model plane, 
wherein a rendering of the three-dimensional image model is manipulated according to the pose 
of the model plane. 

Regarding dependent claim 53, Harrington discloses the information analysis and image 
preparation need be accomplished in essentially real-time for the image display to appear as real 
as possible to the user (column 5, lines 32 - 35), which reads on the claimed wherein the 
augmented reality video is produced in real time. 

Claims 49 and 57 rejected under 35 U.S.C. 102(f) because the applicant did not invent the 
claimed subject matter. 

Regarding dependent claims 49 and 57, Navab discloses FIG. 6 shows a color cube 
augmented on top of the model plane using OpenGL rendering with a fake shadow in accordance 
with the principles of the present invention (paragraph 12). Navab did not disclose any other 
mention of providing shades or shadows in the disclosed specification. Applicant admitted that 
the fake shadow was made using OpenGL. 

Claim Rejections - 35 USC § 103 
The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
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having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

Claims 49 and 57 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Harrington (US 6,898,307) in view of Navab et al. (US AN 09/915650). 

Regarding dependent claims 49 and 57, Harrington did not expressly discloses adding 
shadow to the three-dimensional image data model on the model plane. Navab discloses FIG. 6 
shows a color cube augmented on top of the model plane using OpenGL rendering with a fake 
shadow in accordance with the principles of the present invention (paragraph 12). It would have 
been obvious for one of ordinary skill in the art at the time of the invention to modify 
Harrington's system by using OpenGL to incorporate fake shadow. One would be motivated to 
do so to provide realistic effects in an Augmented Reality environment. 

Claims 50, 51, 58, 59, 61, and 62 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Harrington (US 6,898,307) in view of Rhoads et al. (US 7,050,603). 

Regarding to independent claim 62, Harrington discloses a camera 12 that captures the 
normal field of view of the user of the headset 10 that is substantially what the user would see 
without the headset 10 (column 4, lines 4 - 24) and the analog signal from the camera 12 is 
converted to a digitized frame that can be stored in the computer 18 for analysis (column 4, lines 
25-38) and the computer 18 must interpret the captured video frames to locate a piece of paper 
within the camera view (column 4, lines 38-67 and column 5, lines I - 34) and the preselected 
movable real object comprises a reference panel such as a screen, tablet or piece of paper and the 
identifying includes recognizing a comer of the panel (claim 2), which reads on the claimed 
providing video data comprising images of a moving model plane having markers. It is inherent 
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that a piece a paper, screen, or tablet is substantially rectangular, which reads on the claimed 
model plane is a substantially rectangular plane. Harrington discloses marker is a comer of the 
panel (claim 2) and a piece of paper can be identified by its white color against a darker 
background (column 4, line 55 - column 5, line 3), which reads on the claimed marker is a 
graphic disposed on an upper surface thereof Harrington further discloses the unprojection 
module 82 that projects three-dimension object onto a two-dimensional viewing plane, where a 
two dimensional image results from the three dimensional object (column 7, lines 28 - 53), 
which reads on the claimed providing a three-dimensional image data model of a product. 
Harrington did not expressly disclose the three-dimensional image data model is an animation. 
Rhoads discloses watermarks may be embedded in two dimensional image renderings or still or 
animated 3D graphical objects and the embedded object can be composited with a video stream 
to form a video program, such as a movie or television program and graphical objects that link to 
information or electronic commerce transactions can be added to a video product, such as a 
movie, when its converted from one format to another (column 18, lines 5-18). It would have 
been obvious for one of ordinary skill in the art at the time of the invention to modify 
Harrington's system by projected 3D animated objects as expressed in Rhoads. One would be 
motivated to do so because 3D animated objects are used in many commonly used applications 
such as games (column 1 8, line 63 - column 19, line 5) or interactive shopping (column 20, lines 
41 - 48) that provide users a more convincing virtual environment. Harrington further discloses 
the unprojection step comprises the mathematical calculation comprising taking a two- 
dimensional image and determining what three-dimensional object could have created that two- 
dimensional image with respect to what the object can be and where it could have been 
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positioned and the subject invention utilizes known dimensions with regard to the camera system 
and viewing plane, in combination with assumed characteristics of the piece of paper (column 7, 
Unes 28 - 53), which reads on the claimed determining a pose of the moving model plane 
according to the markers in the video data and the claimed determining an image correspondence 
between the moving model plane having markers and the three-dimensional image data model. 
Harrington discloses the camera provide the computer with the same view that the user sees so 
that the synthetic image is displayed or superimpose in a manner for comfortable and convenient 
interaction between the real reference item within the field of view that the user may be holding 
and manipulating (column 4, lines 4 - 37), which reads on the claimed producing the augmented 
reality video by rendering the three-dimensional image data model of the product superimposed 
on the moving model plane in the video data. Harrington discloses the user could select as the 
application image a particular internet browser engine which, based upon the located orientation 
of the piece of paper, would be transformed to exactly match the detected orientation of the piece 
of paper so that the displayed image provides the illusion of the web page actually appearing on 
the blank piece of paper being held by the user and once the position of the paper page is 
determined, information can be used to correctly distort the application's image so it overlays the 
paper in the eyes of the user (column 5, lines 4 - 35) and the unprojection step comprises the 
mathematical calculation comprising taking a two-dimensional image and determining what 
three-dimensional object could have created that two-dimensional image with respect to what the 
object can be and where it could have been positioned and the subject invention utilizes known 
dimensions with regard to the camera system and viewing plane, in combination with assumed 
characteristics of the piece of paper (column 7, lines 28 - 53), which reads on the claimed 
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rendering of the three-dimensional image data model has substantially the pose of the moving 
model plane. Harrington discloses video capture hardware that converts the analog signal from 
the camera into a digitized frame that can be stored in the computer and the hardware outpus a 
signal to the computer as a digitized and processible representation of what the user sees from 
the camera's field of view and the video generation hardware takes a bit map of pixel values 
from the computer and converts them into a television format that can be displayed on the 
headset and where the signal generated by the camera is different from the signal displayed on 
the headset by the addition of the computer-generated electronic image to thereby form a 
synthetic image as an augmented-reality display (column 4, lines 25 - 38), which reads on the 
claimed converting the augmented reality video into a streaming video format and the claimed 
streaming the augmented reality video having the streaming video format over a communications 
network to a computer for displaying the augmented reality video. Harrington did not expressly 
discloses encoding hyperlink information into the augmented reality video and accessing 
information about the corresponding product by selecting the corresponding product in the 
rendered three-dimensional form from the augmented reality video. Rhoads discloses watermark 
may carry information or programmatic action or link to external information or an action, such 
as retrieval and output of information stored elsewhere in a database, website, etc (column 4, 
lines 31-39) and watermark can be embedded immediately into a video object layer after a 
graphical model is rendered to the video object layer (column 4, lines 40 - 49) and users 
selecting a video object, retrieving the linked information or actions for the selected object and 
rendering them on its user interface (column 16, lines 16-38). It would have been obvious for 
one of ordinary skill in the art at the time of the invention to further modify Harrington's and 
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Rhoads' systems by embedding hyperlink information into 3D rendered objects when selected. 
One would be motivated to do so because this gives users the convenience of retrieving extra 
information about the selected desired object from an external source upon request. 

Regarding independent claim 61 and dependent claim 50, 51, 58, and 59, claims 50, 51, 
58, and 59 are similar in scope as to claim 61, thus the rejection for claim 61 hereinabove is 
applicable to claims 50, 51, 58, and 59. 

Conclusion 

Applicant's amendment necessitated the new ground(s) of rejection presented in this 
Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). 
Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the date of this 
final action. 

The prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure. Zhang, Z., "Flexible Camera Calibration by Viewing a Plane from Unknown 
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Orientations", Proceedings of the Seventh International Conference on Computer Vision, 1999, 
666-673 discloses scaling and caHbration algorithm. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Jeffrey J. Chow whose telephone number is (571)272-8078. The 
examiner can normally be reached on Monday - Friday 10:00AM - 5:00PM (EST). 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Ulka Chauhan can be reached on (571)-272-7782. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would 
like assistance from a USPTO Customer Service Representative or access to the automated 
information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000, 





ULKA CHAUHAN 
SUPERVISORY PATENT EXAMINER 



